Augmented Reality Viewing and Tagging For Medical Procedures

ABSTRACT

Technology is described for augmenting medical imaging for use in a medical procedure. The method can include the operation of receiving an image of patient anatomy captured by a visual image camera during the medical procedure. An acquired medical image associated with the patient anatomy can then be retrieved. Another operation can be associating the acquired medical image to the patient anatomy. An augmentation tag associated with a location in one layer of the acquired medical image can be retrieved. A further operation can be projecting the acquired medical image and the augmentation tag using an augmented reality headset to form a single graphical view as an overlay to the patient anatomy in either 2D, 3D or holographic form.

PRIORITY DATA

This application is a continuation of 16/486,818, filed Aug. 16, 2019,which is a 371 Nationalization of International Application No.PCT/US2018/019024, filed Feb. 21, 2018, which claims benefit of U.S.patent application Ser. No. 15/438,715, filed Feb. 21, 2017, all ofwhich are incorporated herein by reference.

BACKGROUND

Mixed or augmented reality is an emerging area of computing technologywhere images from the physical world and virtual computing worlds may becombined into a mixed reality world. Mixed reality may encompass a widerange of technological combinations that previously were considered tobe only physical reality or only virtual reality. In mixed reality,people, places, and objects from physical and virtual worlds mergetogether in a blended environment. A mixed reality experience may beenabled through existing commercial operating systems or customoperating systems along with the use of a compatible VR (virtualreality) or AR (augmented reality) headset

Augmented reality (AR) is an example of mixed reality where a livedirect view or an indirect view of a physical, real-world environment isaugmented or supplemented by computer-generated sensory input such assound, video, graphics or even global positioning data. As a result, thetechnology can enhance a viewer's current perception of reality.Augmentation is conventionally performed as a real world location isbeing viewed and in semantic context with environmental elements. Withthe help of advanced AR technology (e.g. adding computer vision andobject recognition) the information about the surrounding real world ofthe user becomes interactive and may be digitally manipulated.Information about the environment and its objects may be overlaid on thereal world. This information overlaid on the real world can be virtualimages or real information. Augmented reality can bring the componentsof the digital world into a person's perceived real world.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an illustration of an example use of an augmented realityhead set to augment a medical procedure with acquired medical images.

FIG. 1B is an illustration of an example use of a three dimensionalaugmentation tag for a spleen.

FIG. 1C is an illustration of an example use of a plurality ofaugmentation tags.

FIG. 1D is an illustration of two dimensional augmentation tags that areautomatically made into three dimensional augmentation tags.

FIG. 1E illustrates a two dimensional augmentation tag for a kidney.

FIG. 1F illustrates a three dimensional augmentation tag for a kidney.

FIG. 1G illustrates a three dimensional augmentation tag for an abscess.

FIG. 1H illustrates a three dimensional augmentation tag for an abscessas covered by imaged skin.

FIGS. 1I and 1J illustrate a three dimensional augmentation tag for ananatomical structure in a colon.

FIGS. 2A-2D illustrate an augmentation tag in an acquired medical imageused as an overlay for an appendix.

FIG. 3A illustrates an acquired medical image that is overlaid on apatient's head and depicts a cross section of the patient's neck.

FIG. 3B illustrates an acquired medical image that is overlaid on apatient's head region and illustrates that hand gestures may be used tocontrol the augmented reality headset.

FIG. 4 is an illustration of an example use of an augmented reality headset to augment a medical procedure using physical patient identifiers.

FIG. 5 is a block diagram illustrating an example of a system foraugmenting a view of a medical procedure using data pulled from datastorage.

FIGS. 6 and 7 are flow charts illustrating methods for augmenting a viewof a medical procedure.

FIG. 8 is a block diagram that provides an example illustration of acomputing device that may be employed in the present technology.

DETAILED DESCRIPTION

This technology can provide a system and method for using mixed realityor augmented reality devices to improve surgical, interventionalradiologic, cardiac, or other medical procedures. An augmented realitydevice, such as an augmented reality (AR) headset, may be used tooverlay acquired medical images or virtual images onto a real worldscene (e.g., what user is viewing in the real world) using imagesprojected on lenses or a screen that is partially transparent. In thecase of surgical procedures, already acquired medical images may be usedas overlay images for viewable patient anatomy. These acquired medicalimages may be MRI (magnetic resonance imaging), fluoroscopy, CT(computed tomography), sonography, nuclear medicine, computer generatedimages (CGI), photos, video, or other types of acquired or synthesizedmedical images. An example of synthesized medical images would be ananatomic atlas overlay on the patient's anatomy to guide surgery. Theacquired medical images can be projected onto the lenses of the ARheadset in a location that orients the acquired medical images over anappropriate portion of the patient anatomy of the patient being treated.

The overlay of an acquired medical image can assist a surgeon, doctor orother medical professionals in more accurately performing a treatment oroperation upon a patient's anatomy. One issue that faces doctors andother medical professionals when performing operations upon patients ismaking sure the correct patient anatomy is being operated upon. If thewrong person, wrong appendage or wrong location is being operated, onthen a poor outcome may be the result. The present technology providesaugmented reality tags to more accurately identify patient anatomicalstructures. Patient identifiers (e.g., physical identifiers) that areaffixed to the patient may also be captured and the patient identifiersmay be used to load related acquired medical images and augmentedreality tags for a patient procedure. Examples would be bar codes, QRcodes, patient IDs, physical tags, etc. Morphometric data can also becaptured from viewing of a patient's anatomy in the real world and thenthe morphometric data can be compared with stored morphometric data foran appendage or anatomical structure to be treated. When themorphometric data matches, then the treatment may proceed. Otherwise anerror may be flagged based on the morphometric data. Either of thesemethods (e.g., patient identifiers or morphometric data) can be used touniquely identify the patient and the area of anatomy to be treated.

FIG. 1A illustrates a doctor or medical professional 110 who may beusing an AR headset 112. In one example the AR headset 112 may enable amedical professional to see through lenses where an augmented realityimage or acquired medical image is projected on the semi-transparentlenses. As a more specific example, an MRI image may be overlaid in anarea where an operation is going to occur. The overlaid image results ina compositing of views with a real world scene that is being viewedthrough the semi-transparent lenses or a semi-transparent opticalimaging area. A facet of this ability to overlay images is the camerasthat exist in the augmented reality headset to create a contextual mapof the space in which the patient lies and allows 3D patient data suchas a hologram of images obtained prior to surgery to be merged with thereal world of patient anatomy.

In order to augment a medical procedure or surgery with an augmentedmedical image, a medical doctor or other medical professionals may setup an AR headset 112 in a surgery area 102. A processor associated withthe AR headset 112 may receive an image of patient anatomy 116 using avisual image camera in the AR headset 112. An acquired medical image 114associated with the patient anatomy can then be retrieved. This acquiredmedical image may be an MRI or CT image that is fed via a wirelessconnection to the processors and memory in the AR headset.

The acquired medical image 114 can then be associated with or anchoredto the patient anatomy 116 that has been identified by the AR headset112 in the real world scene or space. The anchoring may include fixingor anchoring the acquired medical image or other related virtual images(including augmentation controls) to a fixed point in the viewable realworld that has been registered by the AR headset 112. Later when adoctor moves view point positions, the acquired medical image 114 canremain fixed in the correct spot with respect to the patient's anatomyand does not move around in the doctor's vision.

An augmentation tag 118 associated with a location in one layer of theacquired medical image 114 or radiological image can also be retrieved.The augmentation tag may be configured to conform to a three dimensional(3D) structure in the acquired medical image 114 or radiological imageto identify an anatomical structure associated with a medical procedureto be performed. The augmentation tag may be a simple geometric shapesuch as a circle, square, triangle or another more complex shape in twodimensions or three dimensions, such as an outline of the anatomy inquestion.

FIG. 1B illustrates that the augmentation tags may be used to pre-labelthe acquired medical image. If a doctor is going to operate on apatient' s spleen 170, then the doctor can apply an augmentation tag tothe patient's spleen during diagnosis of a medical condition (e.g.,splenomegly or another spleen condition). The augmentation tag may covera portion of the patient's anatomical structure as loosely marked by thedoctor.

Then the augmentation processor can then detect the edges of theanatomical structure or use machine pattern recognition and expand orcontract the augmentation tag to conform the augmentation tag to theanatomical structure. Later, when the medical procedure, diagnosis oroperation is taking place, the doctor can view the tag in the shape ofthe anatomical structure (e.g., spleen) and be more certain about thepositive identification of the structure for treatment.

FIG. 1C illustrates that the augmentation tags can be provided in aplurality of anatomical layers of the acquired medical image. Forexample, the augmentation tags can identify different structures in eachlayer of the acquired medical images and varying procedures that shouldoccur at those separate layers in the patient's anatomy. In addition,the augmentation tags can be associated to form a tag group 130 a-e toguide a surgeon. As a more specific example, there may be a group ofdiseased anatomical structures that needs to be operated on and thesestructures can be linked together in an ordered group and the surgeoncan be guided from one augmentation tag to the next. This avoids missingany of the anatomy that needs to be operated on or skipping any of theimportant procedure. Similarly, an ordering may be provided for asurgeon to address different structures in a selected order in order toassist with remembering of any unusual orderings.

In another example, the augmentation tags may guide a surgeon through asafe path to avoid structures, such as an artery 120, which may bedelicate or cause problems if the structures are damaged. This way thesurgeon can have a mapped path for an operation before the operationbegins. A plurality of augmentation tags that are linked together may beused to show a surgical path through the patient's anatomy. For example,the tags may be used to show a surgeon where the surgeon has planned tocut as the surgeon passes through multiple layers. The augmentation tagsmay also show a surgeon where the surgeon has planned to cut in alateral direction or travel with an endoscope. This allows to thesurgeon to plan out a surgery route in advance and follow that routemore accurately during the surgery when using the augmentation tags andthe acquired medical image(s). This may reduce the risk of the surgeryand improve the overall patient outcome.

In some cases, a plurality of augmentation tags may be linked togetherto represent one anatomical structure which crosses separate layers ofthe acquired medical image or radiological image. Thus, each tag mayidentify that anatomical structure in a separate layer.

Returning to FIG. 1A, the acquired medical image 114 and theaugmentation tag(s) 118 can then be projected onto the lenses of theaugmented reality headset 112 to form a single graphical view for themedical professional(s) wearing the AR headset 112. As a result, theacquired medical image 114 and augmentation tag 118 may appear as thoughthe images are overlaid directly on the patient anatomy.

In an alternative configuration, the augmentation tag may have a tagextender 122 (FIG. 1B) that can be a geometric shape that extends awayfrom the augmentation tag. Examples of the geometric shape may be aflag, polygon, rectangular box, pyramid or other extension that maydisplay information to a surgeon. The information that may be displayedin the tag extender may identify the procedure to be performed on theanatomy, identify the patient, identify pre-existing patient risks,identify drug allergies, identify known complications or other data thatis not directly related to the patient's anatomy but is related to thepatient or medical procedure.

Referring now to FIG. 1D, in one further configuration of theaugmentation tags, a doctor may create one or more 2D (two dimensional)augmentation tags 150 a-b each located on one of a plurality of layersof the acquired medical image. The plurality of 2D (two dimensional)augmentation tags can be automatically joined together by an applicationto form a 3D augmentation tag or 3D shape that extends through multiplelayers of the acquired medical image. For example, a method can beconfigured to create a 3D shape 160 from the 2D tags. Similarly, the 2Dimage or one slice of the acquired medical image can be annotated withone or more 2D images, and then the 2D annotations can be expanded to a3D shape using known 3D shapes such as lines, rectangles, squares,spheres, regular polygons, irregular polygons, or other shapesselectable from a menu. Marking a 3D acquired medical image can bechallenging because of the many layers in such images but enabling 2Dmarking that can be automatically turned into 3D images makes marking ofacquired medical images easier for medical professionals. Also, apartial outline of a structure in a patient's anatomy of interest may bedrawn and then an application may identify the rest of the anatomicalstructure using edge and feature detection. This automatic objectdetection enables a full outline or full shape, as detected by anapplication, to be used to identify the patient's anatomy to be treated.For example, an organ to be resected can be highlighted with eithermanual technique, anatomic edge detection or neural networks where thesystem is trained to find the organ.

The acquired medical image may be a radiological image that is acquiredusing medical imaging techniques such as a MRI (magnetic resonanceimaging), magnetic resonance angiography (MRA), fMRI (function magneticresonance imaging), mammography, CT (computed tomography), fluoroscopy,X-rays, nuclear medicine (e.g., bone scans, thyroid scans, etc.), PET(positron emission tomography), ultrasound images or any other medicalimaging techniques. The acquired medical image may also be an imagecreated using photographic images internal to a human body such as acolonoscopy, virtual colonoscopy endoscopy, or arthroscopy. In addition,the acquired medical images may include completely virtual elements thathave been graphically rendered in a virtual 3D space and added into theacquired medical image. This may include rendered tags, rendered anatomy(e.g., rendered bones, nerves, tendons, etc.), rendered orthopedichardware to be installed, or other similar virtual renderings.

FIG. 1E illustrates a two dimensional (2D) augmentation tag for akidney. In this case, a medical professional may annotate a 2D slice orlayer of an acquired medical image to create an augmentation tag 172.The medical professional may notate the image by carefully outlining thekidney or by selecting a point in the center mass of the kidney tissueand requesting an application to find the boundary of the kidney. This2D augmentation tag can be turned into a three dimensional (3D)augmentation tag is described in this disclosure. In an alternativeconfiguration, FIG. 1E can illustrate a 2D cross-sectional view of a 3Daugmentation tag.

FIG. 1F further illustrates a three dimensional (3D) augmentation tagfor a kidney 174. The 3D augmentation tag can be created by a medicalprofessional requesting the application to take the 2D augmentation tagand find the entire kidney shape by identifying tissue similar to kidneytissue selected by a medical professional in the acquired medical image.Once the application has identified what is believed by the applicationto be the kidney, the medical professional may change the shape of thekidney to most accurately identify what needs to be addressed by amedical procedure. This may mean adjusting the augmentation tag toidentify protrusions or areas that were not correctly captured by theapplication or excluding portions that are not desired to be in theaugmentation tag. Alternatively, the medical professional can annotatethe kidney entirely in 3D, if desired. The volume of the 3D augmentationtag can be computed in order to assist a medical professional withdetermining how large or small the anatomical structure.

FIG. 1G illustrates a three dimensional (3D) augmentation tag for anabscess 176 where the 3D augmentation tag is shown with the acquiredmedical image that depicts a bowel structure and bone structure of thepatient, where the abscess is under the bowel structure. In the acquiredmedical image, the bowel structure may be turned on or off so that amedical professional can see the augmentation tag in various contexts(e.g. with or without bowels). FIG. 1H further illustrates a 3Daugmentation tag for an abscess as covered by imaged skin. This viewusing the imaged skin can provide a different context for the medicalprofessionals who may need to perform a medical procedure. As discussed,these acquired medical images with the augmentation tags can be used asoverlays using an AR headset.

FIGS. 11 and 1J illustrate a 3D augmentation tag for an anatomicalstructure (e.g., a polyp or tumor) in a colon. The image of the colonmay be created using neural network machine learning methods. Polyps orpossible cancerous structures in the colon can be identified usingneural network machine learning techniques. The neural networks may havebeen trained using a large number of medical training data cases to beable to find or classify the polyp. Alternatively, a medicalprofessional can mark the polyp with an augmentation tag or useautomatic assistance of an application to more easily mark the polyp.

FIG. 2A illustrates an augmentation tag 210 and an acquired medicalimage 212 used as an overlay with an AR headset for an appendix relatedmedical procedure. The augmentation tag 210 and acquired medical image212 are viewed from a right side of a patient. The acquired medicalimage 212 also illustrates skeletal structure and other patient organs.While the augmentation tag is illustrated in this figure as darkenedsemi-spherical structure (or in some related figures as a lighterstructure), other augmentation tags may be used that are colored (e.g.,green, red, blue, etc.) or textured (e.g., special graphical texturessuch as hatching, striping, or other textures). FIG. 2B illustrates thesame augmentation tag 210 and acquired medical image as viewed from theleft side of the patient. FIG. 2C illustrates an augmentation tag 210for the appendix as viewed from the right side of a patient and anoverlay from the acquired medical image that is a cross sectional viewin a partial sagittal plane from the acquired medical image. FIG. 3Dillustrates an augmentation tag 210 and an acquired medical image thatis a cross sectional view in a transverse plane for the acquired medicalimage.

FIG. 3A illustrates the use of an augmented reality headset where anacquired medical image for a patient's head and brain is anchored to thereal world view of the patient's head. As can be seen in the image, across section of the patient's brain is visible in the overlay on thepatient's head and neck and medical personal can view anatomicalstructure of the patient prior to any invasive action on the part of themedical professionals.

FIG. 3B illustrates that hand gestures may be in front of the AR headsetin order to control the AR headset. For example, zooming or shrinkingmotions can be made by opening and closing a hand. Similarly, movingthrough multiple layers of the acquired medical image can be performedby pinching each side of the acquired medical image and simultaneouslymoving hands up or down.

This technology may use augmented reality (AR) headsets, virtual realityVR headsets, transparent overlay systems that are not worn on the head,or other types of mixed reality systems that are available or may becomeavailable. To further help explain this technology, an example of an ARheadset will be described now. However, the use of this technology isnot limited to only AR headsets but may use other types of augmentedreality or mixed reality systems. An example AR headset may have a thickvisor-like band that wraps around the head of medical professionals. Thevisor may be lightweight and adjustable to fit different head sizes. Insome configurations, the AR headset may be wirelessly networked andprovide audio output. Alternatively, medical professionals may wear theAR headset like corrective glasses and the processor portion of the ARheadset may be worn on around a person's waist, or attached to aseparate processing station.

The AR headset may include holographic lenses and a depth camera as wellas speakers above the ears and an on-board processor with at least oneGPU (graphics processing unit). The headset may also include venting tokeep the headset from overheating. The wireless connection may beBluetooth, Wi-Fi, ZigBee, cellular, or another type of wirelessconnection.

Additional support sensors may be included in an AR headset such as anambient light sensor and multiple environment sensing cameras that workin combination with a depth sensing camera to identify a work spacewithin which the augmented reality environment can be created.

A high definition camera may also be included in an AR headset to takephotos and record mixed reality video. Multiple microphones may beincluded to capture local audio. The AR headset may include are-chargeable battery system or the AR headset may receive power from anadapter in a laptop fashion.

Some AR headsets may be controlled by gestures, voice or a remotecontroller and such controls may be used to control objects orinterfaces of the AR headsets. Hand gestures may also be made in frontof the AR headset and may be captured by one or more of the front facingcameras. In addition, a pointer device, such as a mouse, stylus, clickeror other pointer devices may be used to control the AR headset.

The depth camera may be low power and may have a field of view of 60degrees in the X axis and 60 degrees in the Y axis, for example. Otherfields of view for a depth camera can be used. There may be multiplecameras around the headset (i.e., on the front and sides). These camerascan capture video of the physical surroundings, track the user's handsto identify gestures, and help to track head movements along with motionsensors on the AR headset.

The ability to blend acquired medical images, virtual models, andaugmentation tags with a live environment means that the lenses may betransparent. In one configuration that may be two lenses, one for eacheye, and the lenses may be made up of three layers of glass (blue, greenand red). A projection engine or light engine above the lenses mayproject light into the headset and tiny corrugated grooves in each layerof glass diffract these light particles, making the light bounce aroundto enhance the illusion of perceiving virtual objects at virtualdistances. These images may appear as what may be called holograms. Oncethe AR headset has mapped a room, the virtual images can be blended intothe real environment viewed through the semi-transparent lenses.

The AR headset can anchor or “pin” virtual images or objects into placewith respect to the real environment or room. Once a virtual object orvirtual image is locked in place for the viewable environment or realenvironment, then a user can move around the virtual object or virtualimage to view the virtual object from different angles without theobject or overlay image moving.

FIG. 4 illustrates the augmentation of a view of a medical procedure 402for medical professionals 410 using an AR headset 412. A live image ofpatient anatomy and the surrounding environment or room may be obtainedusing a live image camera of the AR headset 412. A patient marker 420that is located on the patient and can be identified in the image orvideo of the patient anatomy as obtained by the AR headset. The patientmarker 420 may include information identifying the patient, anatomy tobe operated upon, a patient orientation marker 422, and/or an imageinversion prevention tag 422. The patient orientation marker 422 and theimage inversion prevention tag 422 may be separate or combined into onemarker or tag with each other or the patient marker 420.

A visually scannable symbol attached to a patient, such as a barcode,can provide information to retrieve the identity of the patient and toretrieve an associated acquired medical image 414 or multiple associatedacquired images. The patient orientation marker 422 can be used toensure that the acquired medical image is oriented correctly withrespect to the patient anatomy. The patient marker 420 and an imageorientation marker 422 in the radiological image may be matched withmarkers in the acquired medical image to enable matching orientation ofthe acquired medical image with the patient orientation marker 422 onthe patient anatomy. This avoids orientation or calibration errors whenthe acquired medical image 414 is aligned with the patient anatomy. Theimage inversion tag 422 can be used to make sure the image is notinverted or flipped over when the alignment occurs.

In one configuration of the technology, a bar code, QR code, or specialmarker may be used to mark the right patient for the medical procedure,mark right side of the patient, or mark the right appendage on which toperform a medical procedure.

The optically scannable code or marker on the patient's anatomy can alsobe compared the patient data associated with the acquired medical imageto see if the patient's acquired medical image and/or anatomy matcheswith the patient marker 420 provided.

An acquired medical image 414 can be associated with the patient anatomy416 based in part on the patient marker 420. This means that the correctacquired medical image 414 can be retrieved based on the patientidentity and the anatomy to be operated on, as identified in advance.The acquired medical image 414 can be anchored or fixed to the patientanatomy based in part on the patient orientation marker 422 or imageinversion prevention tag 422. Alternatively, the acquired medical image414 can be overlaid onto the patient anatomy by identifying thetopological structure of the patient anatomy 416. Automating thisprocess can reduce errors, and vastly reduce expensive operating roomtime.

An augmentation tag 418 associated with the patient marker 420 and alocation in the acquired medical image or radiological image can also beretrieved for enabling viewing of an augmentation tag 418 overlaid ontothe patient's anatomy 416. The acquired medical image 414 and theaugmentation tag 418 can be projected onto lenses in an augmentedreality headset 412 to form a single graphical view which is virtuallyoverlaid onto the patient anatomy being viewed.

In one configuration, an endoscopic video feed may be combined into thesingle graphic view. An augmentation tag can also be used to identify asize and shape of 3D structures in the endoscopic video feed and toidentify where an endoscope is estimated to be located in the patientanatomy.

FIG. 5 illustrates a system for augmenting a view of a medicalprocedure, such as a surgical procedure. The system may include a camerain an AR headset 542 or a camera associated with the system, and thecamera may be configured to obtain a live image of patient anatomy 510.The camera may provide a live video feed of viewable patient anatomy foranalysis by the AR headset 542. The video may be analyzed to identifyenvironment orientation cues such as appendage locations, edgedetection, depth detection, or features in the video images captured. Inaddition, the live video may be recorded or otherwise streamed toanother location.

An augmentation processor 520 in the system can be configured to measureor define morphometric measurements of patient anatomy 510 as capturedby the camera. The morphometric measurements may be metrics about thepatient's anatomy, such as: shape, width, height, depth and contour ofan appendage or patient anatomy. These morphometric measurements may berelative measurements or absolute measurements where a reference objector measurement key is provided. The augmented reality system can map theexternal contours of the patient using the depth camera or distanceranging cameras and can create a polygonal mesh that can be compared tothe surface layer of the patient's 3D data obtained with imagingtechniques. Thus both the “virtual” and “real” data can be compared fora match much like a finger print in 3D.

The augmentation processor 520 may determine whether the morphometricmeasurements measured from live video or images match pre-measuredmorphometric measurements associated with the patient anatomy asretrieved using the patient marker 530. For example, a patient's leg aspresented in person may have certain measurements for length, width,height, shape or other morphmetric measures that can be matched topre-measured morphometric measurements.

An acquired medical image 524 associated with the patient anatomy canthen be retrieved as defined by the patient marker 530 and the patientanatomy 510. The acquired medical image 540 can be aligned with thepatient anatomy 510 using the morphometric measurements and confirmedmorphometric measurements.

The system may include an AR headset 542 that is configured to projectthe acquired medical image 522 and the augmentation tag(s) 524 ontosemi-transparent lenses in the AR headset to form a single graphicalview which is overlaid 540 on the patient anatomy being viewed by theuser or medical professional using the AR headset. For example, asimilar image may be projected into each eye, which may enable a 3D(three dimensional) viewing effect.

A notification can also be provided to the doctor or medicalprofessional that the acquired medical image matches the patient anatomyas defined by the morphometric measurements. This way the doctor can beassured that the right patient and correct physical structure (correctleg or arm, etc.) is being operated upon with the correct acquiredmedical image 522. Not only is the patient's identity being checked by apatient marker but information or images retrieved can be checked usinga comparison of the real patient anatomy being view in the real world ascompared the morphometric measurements previously made. When a match isconfirmed, this increases the chances the medical procedure will beaccurate or correct.

In the case, where the morphometric measurements from the patient in themedical procedure do not match with the previously collectedmorphometric measurements associated with the patient identifier, thenthen a warning or error can be presented to the doctor or medicalprofessionals. If a matching error does occur, the doctor may determinewhat information needs to be checked further and whether to proceed withthe medical procedure or to terminate the medical procedure. Thisconfirmation may be especially important where the patient is undersedation and cannot communicate with the attending medicalprofessionals.

The morphometric analysis or capture for the patient's anatomy duringimaging to identify a medical problem, and then the checking of themorphometric data upon treatment of the problem can result in moreaccurate patient treatment and avoid possible errors. As a result, thistechnology may perform a morphometric memorization of the patient or ofanatomical aspects of the patient that are desired to be recorded. Thena graphical warning can be provided to medical professionals, if themorphometric comparison of the pre-captured anatomical structure and theanatomical structure currently viewed using the AR headset during amedical procedure does not match.

For example, if a doctor is going to operate on a patient's skull andthe doctor needs to put a burr hole in the skull, then the doctor canhave the augmented reality surgery system check the patient to make surethe shape of that patient's skull in the medical procedure matches thesame shape as the information in the acquired medical image that waspreviously captured during a diagnosis phase. If the shape of the skullis not the same, a warning can be flashed to the doctor or medicalpersonnel. Certain metrics for anatomical structure, such as the size,shape, amount of fat, facial details and similar skull details can bechecked against morphometric data stored that previously measured thesedetails for the patient.

In the context of morphometric analysis, a marking system with a barcode can be scanned to identify the patient. For example, an ID band maybe scanned and the ID information may activate a download of thepatient's acquired medical images and morphometric data because of thebar code. The acquisition of the patients acquired medical images withmorphometric data associated with the acquired medical images can thenbe checked against the morphometric data computed from the live imagesor medical procedure images being captured from the patient using the ARheadset. Once the patient identity has been confirmed using morphometry,then augmented tags may be loaded and combined with the acquired medicalimage. Finally, the surgeon may begin a medical procedure on the patientbecause the patient identity, acquired medical image, morphometry andsimilar aspects of the surgery have been verified.

The patient marker 530, as discussed above, may be identified in theimage of the patient anatomy 510 and the patient marker 530 may includeinformation identifying the patient and pre-measured morphometricmeasurements 526 stored in a database. The patient marker 530 may be ascannable symbol including at least one of: a 1D (one dimensional) barcode, a 2D (two dimensional) bar code, a picture, an image, a customgenerated geometric shape, or a RFID (radio frequency ID) andinformation from the patient marker 530 may be used to retrieve theidentity of the patient, patient information 528, and to retrieve theacquired medical image 522. In a more specific examples, the patientmarker 530 may have a database key or value used to look-up or query forthe pre-measured morphometric measurements 526 stored in separatedatabase.

When the patient identity has been obtained using visual scanning, abracelet, RFID or morphmetry, then an acquired medical image 522matching the patient identity and other patient information may beloaded along with augmentation tags that match the identity and theacquired medical image. These augmentation tags may be loaded, orientedbased on the identity of the patient determined with the identity tagand/or morphometry.

The system may also retrieve an augmentation tag associated with apatient orientation marker 532 and a location (e.g., a Cartesiancoordinate or another relative coordinate in the acquired medical image)in the acquired medical image or radiological image. The images of theacquired medical image and the augmentation tag may be combined into asingle view using the patient marker and location. In addition, thepatient orientation marker 532 may be used or matched with an imageorientation tag in the acquired medical image or radiological image toorient the acquired medical image correctly over the patient's anatomy.

The system that includes the augmentation processor may also perform anoperation identifying an image inversion prevention tag 532. An acquiredmedical image and patient anatomy can be aligned (e.g., rotated, flippedor inverted) using an image inversion prevention tag to ensure theacquired medical image is oriented correctly with respect to the patientanatomy.

In an alternative example of the present system, the headset may be avirtual reality (VR) headset, into which a video feed of a patient andoperating room is fed and the acquired medical image can be compositedtogether with the video feed. In this case the VR headset may not betransparent but the VR headset may enable the composited image to beused for diagnostic or other treatment purposes.

In one example, morphometry can be used to identify procedure roadmarkers during a medical procedure such as endoscopy, arthroscopy,laproscopy, etc. For example, in a colonoscopy, certain uniquely shapedanatomical structures may be marked prior to the medical procedure.These anatomical structures may be marked using augmentation tags. In aspecific example, 5 structures may be marked as road markers ormilestones in the medical procedure. As the actual medical procedureprogresses, the system may match one of the 5 structures viewed througha camera image and show a surgeon where the scope of the surgeon islocated in the patient anatomy based on seeing a physical object thatmatches a virtual object that has already been recorded usingmorphometry. This location can be displayed as an overlay to the patientanatomy or in combination with the acquired medical image. This matchingmay proceed until all 5 structures have been identified during thecourse of the procedure.

In the case of a colonoscopy, many repeating structures exist and so ifthe unusual structures have been tagged in advance, then when the videofeed from the colonoscopy has a match for an anatomical structure thathas been previously tagged, the system can provide the surgeon ormedical professionals doing the colonoscopy with an estimated positionfor where the endoscope is in the colon. This type of estimation cansimilarly apply to other types of scope style surgeries that might occurin the abdomen, joints, lungs, other portions of the gastric system(e.g., small intestine or stomach), reproductive organs, ureter, therespiratory system, etc. This technology can show where the scope orrelated procedure instruments are in the acquired medical image (MRI, CTscan) or in a virtual image based on a distance traveled or a matchedanatomical structure in the acquired medical image.

The system can also provide a multi-feed display. The visual output formedical professionals can display: 1) an image of the visible skin (ascaptured by the visible light camera), 2) an image or video from ascope, 3) a virtual rendered image (CGI or computer generated image), 4)augmentation tags, and/or 5) an acquired medical image (e.g., the X-ray,MRI, or CT image). This allows medical personal to see multiple imageson a single screen instead of multiple screens (e.g., one for the MRI,one on the scope device, one for the medical chart, etc.). This alsoenables medical professionals to switch between different data views.

As referred to herein, a “medical professional” may include physicians,physician assistants, nurse medical professionals, medical specialists,and any of a variety of other types of health care professionals.

As referred to herein, a “medical procedure” may include the science orpractice of the diagnosis, treatment, and prevention of disease. Amedical procedure may encompass a variety of health care practicesintended to maintain and/or restore health by the prevention andtreatment of illness in human beings. A medical procedure may also applyto tasks relating to health science, biomedical research, and medicaltechnology to diagnose and treat injury and disease, such as throughmedication or surgery, as well as through therapies such aspsychotherapy, traction, prostheses, biologics, ionizing radiation andso forth.

While the present technology is described in terms of medicine, thetechnology may alternately be applied in other areas of technology,science, etc. in which productivity is measured, such as according to atype of unit indicative of time, effort, skill, and so forth involved incompleting a task.

FIG. 6 is a flow chart illustrating a method for augmenting medicalimaging for use in a medical procedure. The method may include receivingan image of patient anatomy using a visual image camera, as in block610. The image of the patient anatomy may be collected when the patientis having a medical procedure performed. An acquired medical imageassociated with the patient anatomy may then be retrieved, as in block620. An example of an acquired medical image may be an MM, fMRI or CTscan.

The acquired medical image may then be associated with the patientanatomy, as in block 630. This association may be anchoring or fixing ofthe acquired medical image to a reference point on the patient's anatomythat matches the same anatomical point in the acquired medical image. Anaugmentation tag associated with a location in one layer of the acquiredmedical image can then be retrieved, as in block 640. This augmentationtag may represent an anatomical structure, a point where an incision maybe made, or other mappings and markings for the medical procedure. Theacquired medical image and the augmentation tag can be projected ontolenses in an augmented reality headset to form a single graphical view,as in block 650.

FIG. 7 is a flow chart illustrating a method to augment a view of amedical procedure. The method may include receiving an image of patientanatomy using a live image camera, as in block 710. This may be an imageof a patient after a medical procedure is underway. A patient marker inthe image of the subject matter may be identified, as in block 720. Thepatient marker may include information identifying the patient, anatomyto be operated upon, a patient orientation marker or an image inversionprevention tag.

An acquired medical image associated with the patient anatomy may beretrieved based in part on the patient marker, as in block 730. Theacquired medical image may be a radiological medical image, a computerrendered image, or another type of visual light image. The acquiredmedical image can be anchored to the patient anatomy based in part onthe patient orientation marker, as in block 740.

An augmentation tag associated with the patient marker and a location inthe radiological image may be retrieved, as in block 750. The acquiredmedical image and the augmentation tag may be projected onto lenses inan augmented reality headset to form a single graphical view which isoverlaid on the patient anatomy being viewed, as in block 760.

FIG. 8 illustrates a computing device 810 on which modules of thistechnology may execute. A computing device 810 is illustrated on which ahigh level example of the technology may be executed. The computingdevice 810 may include one or more processors 812 that are incommunication with memory devices 820. The computing device may includea local communication interface 818 for the components in the computingdevice. For example, the local communication interface may be a localdata bus and/or any related address or control busses as may be desired.

The memory device 820 may contain modules 824 that are executable by theprocessor(s) 812 and data for the modules 824. The modules 824 mayexecute the functions described earlier. A data store 822 may also belocated in the memory device 820 for storing data related to the modules824 and other applications along with an operating system that isexecutable by the processor(s) 812.

Other applications may also be stored in the memory device 820 and maybe executable by the processor(s) 812. Components or modules discussedin this description that may be implemented in the form of softwareusing high programming level languages that are compiled, interpreted orexecuted using a hybrid of the methods.

The computing device may also have access to I/O (input/output) devices814 that are usable by the computing devices. An example of an I/Odevice is a display screen that is available to display output from thecomputing devices. Other known I/O device may be used with the computingdevice as desired. Networking devices 816 and similar communicationdevices may be included in the computing device. The networking devices816 may be wired or wireless networking devices that connect to theinternet, a LAN, WAN, or other computing network.

The components or modules that are shown as being stored in the memorydevice 820 may be executed by the processor 812. The term “executable”may mean a program file that is in a form that may be executed by aprocessor 812. For example, a program in a higher level language may becompiled into machine code in a format that may be loaded into a randomaccess portion of the memory device 820 and executed by the processor812, or source code may be loaded by another executable program andinterpreted to generate instructions in a random access portion of thememory to be executed by a processor. The executable program may bestored in any portion or component of the memory device 820. Forexample, the memory device 820 may be random access memory (RAM), readonly memory (ROM), flash memory, a solid state drive, memory card, ahard drive, optical disk, floppy disk, magnetic tape, or any othermemory components.

The processor 812 may represent multiple processors and the memory 820may represent multiple memory units that operate in parallel to theprocessing circuits. This may provide parallel processing channels forthe processes and data in the system. The local interface 818 may beused as a network to facilitate communication between any of themultiple processors and multiple memories. The local interface 818 mayuse additional systems designed for coordinating communication such asload balancing, bulk data transfer, and similar systems.

Some of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more blocks of computer instructions, whichmay be organized as an object, procedure, or function. Nevertheless, theexecutables of an identified module need not be physically locatedtogether, but may comprise disparate instructions stored in differentlocations which comprise the module and achieve the stated purpose forthe module when joined logically together.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices. The modules may bepassive or active, including agents operable to perform desiredfunctions.

The technology described here can also be stored on a computer readablestorage medium that includes volatile and non-volatile, removable andnon-removable media implemented with any technology for the storage ofinformation such as computer readable instructions, data structures,program modules, or other data. Computer readable storage media include,but is not limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tapes, magnetic disk storage orother magnetic storage devices, or any other computer storage mediumwhich can be used to store the desired information and describedtechnology.

The devices described herein may also contain communication connectionsor networking apparatus and networking connections that allow thedevices to communicate with other devices. Communication connections arean example of communication media. Communication media typicallyembodies computer readable instructions, data structures, programmodules and other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. A “modulated data signal” means a signal that has one or more ofits characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency, infrared, and other wireless media. The term computerreadable media as used herein includes communication media.

Reference was made to the examples illustrated in the drawings, andspecific language was used herein to describe the same. It willnevertheless be understood that no limitation of the scope of thetechnology is thereby intended. Alterations and further modifications ofthe features illustrated herein, and additional applications of theexamples as illustrated herein, which would occur to one skilled in therelevant art and having possession of this disclosure, are to beconsidered within the scope of the description.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more examples. In thepreceding description, numerous specific details were provided, such asexamples of various configurations to provide a thorough understandingof examples of the described technology. One skilled in the relevant artwill recognize, however, that the technology can be practiced withoutone or more of the specific details, or with other methods, components,devices, etc. In other instances, well-known structures or operationsare not shown or described in detail to avoid obscuring aspects of thetechnology.

Although the subject matter has been described in language specific tostructural features and/or operations, it is to be understood that thesubject matter defined in the appended claims is not necessarily limitedto the specific features and operations described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims. Numerous modifications and alternativearrangements can be devised without departing from the spirit and scopeof the described technology.

1. A method for augmenting medical imaging of a patient, the medicalimaging displayed using an augmented reality headset worn by a medicalprofessional, the method comprising: retrieving an acquired medicalimage associated with patient anatomy from data storage, the acquiredmedical image comprising imaging acquired of one or more anatomicalstructures of the patient anatomy; aligning the acquired medical imagewith a portion of the patient anatomy viewable through augmented realityheadset, wherein the one or more anatomical structures of the medicalimaging are aligned with the patient anatomy; retrieving an augmentationtag from data storage, the augmentation tag associated with a locationof the acquired medical image, wherein the augmentation tag identifiesat least one anatomical structure of the acquired medical image found atthe location; and projecting the acquired medical image and theaugmentation tag using the augmented reality headset to form a singlegraphical view as an overlay to the patient anatomy viewable through alens of the augmented reality headset.
 2. The method as in claim 1,further wherein augmentation tag conforms to a three dimensional (3D)structure in the acquired medical image to identify an anatomicalstructure associated with a medical procedure to be performed.
 3. Themethod as in claim 1, wherein a plurality of augmentation tags areprovided in a plurality of layers of the acquired medical image and theplurality of augmentation tags are associated to form a tag group toguide a surgeon.
 4. The method as in claim 1, further comprising:creating a plurality of two dimensional (2D) augmentation tags eachlocated on one of a plurality of layers of the acquired medical image;and joining the plurality of 2D augmentation tags to form a threedimensional (3D) augmentation tag that extends through multiple layersof the acquired medical image.
 5. The method as in claim 1, whereinretrieving an augmentation tag further comprises retrieving a pluralityof augmentation tags that are linked together to represent an anatomicalstructure in independent layers of the acquired medical image.
 6. Themethod as in claim 1, wherein retrieving an augmentation tag furthercomprises retrieving a plurality of augmentation tags that are linkedtogether to show a surgical path of an incision through the patientanatomy with respect to the one or more anatomical structures through aplurality of layers of the acquired medical image.
 7. The method as inclaim 1, further comprising capturing the acquired medical image using aMRI, CT scan, X-ray, ultrasound, or photographic images internal to ahuman body.
 8. A non-transitory machine readable storage medium havinginstructions embodied thereon, the instructions when executed cause aprocessor to augment medical imaging of a patient during a medicalprocedure using an augmented reality headset worn by a medicalprofessional during the medical procedure, comprising: identifying apatient marker on patient anatomy viewable through the augmented realityheadset, the patient marker comprising information identifying thepatient, information identifying patient anatomy that is a subject ofthe medical procedure, a patient orientation marker, or an imageinversion prevention tag; retrieving an acquired medical imageassociated with the patient anatomy from a data store based in part onthe patient marker, the acquired medical image comprising imagingacquired of one or more anatomical structures of the patient anatomy;and anchoring the acquired medical image to the patient anatomy based inpart on the patient orientation marker; retrieving an augmentation tagfrom data storage, the augmentation tag associated with the patientmarker and a location in the acquired medical image, the augmentationtag identifying at least one anatomical structure of the acquiredmedical image found at the location; and projecting, during the medicalprocedure, the acquired medical image and the augmentation tag ontolenses in an augmented reality headset to form a single graphical viewwhich is overlaid on the patient anatomy viewable by the medicalprofessional through the lenses of the augmented reality headset.
 9. Thenon-transitory machine readable storage medium as in claim 8, furthercomprising matching the patient orientation marker with an imageorientation marker in the acquired medical image to enable matchingorientation of the acquired medical image with the patient orientationmarker on the patient anatomy.
 10. The non-transitory machine readablestorage medium as in claim 8, further comprising using a visuallyscannable symbol attached to a patient to retrieve an identity of apatient and to enable retrieval of the acquired medical image associatedwith the visually scannable symbol.
 11. The non-transitory machinereadable storage medium as in claim 8, further comprising using theimage inversion prevention tag to ensure the acquired medical image isnot inverted with respect to the patient anatomy.
 12. The non-transitorymachine readable storage medium as in claim 8, further comprising:combining an endoscopic video feed with the single graphical view whichis overlaid on the patient anatomy viewable by the medical professionalthrough the lenses of the augmented reality headset; and using theaugmentation tag to identify a size and shape of 3D structures in theendoscopic video feed and to match where an endoscope is estimated to belocated in the patient anatomy using the augmentation tag.
 13. A systemfor augmenting a view of patient anatomy during a medical procedure fora medical professional, comprising: an augmented reality headset throughwhich to view the patient anatomy during the medical procedure; anaugmentation processor in communication configured to: capturemorphometric measurements of the patient anatomy through the augmentedreality headset during the medical procedure; identify a patient markerof the patient anatomy using the augmented reality headset, the patientmarker comprising information identifying a patient in order to retrievepre-measured morphometric measurements; retrieving pre-measuredmorphometric measurements associated with the patient anatomy from datastorage using the patient marker identified using the augmented realityheadset; determine whether the morphometric measurements of patentanatomy match the pre-measured morphometric measurements associated withthe patient anatomy as retrieved using the patient marker; retrieving anacquired medical image associated with the patient anatomy as defined bythe patient marker and matched morphometric measurements from datastorage, the acquired medical image comprising imaging acquired of oneor more anatomical structures; aligning the acquired medical image withthe view provided by the augmented reality headset during the medicalprocedure of the patient anatomy using the morphometric measurements;and form a single graphical view with the acquired medical image and anaugmentation tag, the augmentation tag identifying at least oneanatomical structure of the acquired medical image; and project thesingle graphical view formed from the acquired medical image and theaugmentation tag onto lenses overlaid on the view of the patient anatomyduring the medical procedure.
 14. The system as in claim 13, wherein themorphometric measurements are shape, width, height, depth and contour ofan appendage or patient anatomy.
 15. The system as in claim 13, whereinthe acquired medical image is a MRI, CT scan, X-ray, ultrasound, orphotographic images internal to a human body.
 16. The system as in claim13, wherein the acquired medical image is displayed using the augmentedreality headset with the acquired medical image projected on asemi-transparent optical imaging area.
 17. The system as in claim 13,further comprising matching a patient orientation marker with an imageorientation tag in the acquired medical image to enable correctorientation of the acquired medical image with the patient anatomy. 18.The system as in claim 13, wherein the patient marker is a scannablesymbol including at least one of: a 1D (one dimensional) bar code, a 2D(two dimensional) bar code, a picture, a custom generated geometricshape, or a RFID (radio frequency ID) used to retrieve an identity ofthe patient and to retrieve the acquired medical image.
 19. The systemas in claim 13, wherein the augmentation processor performs: identifyingan image inversion prevention tag; and aligning the acquired medicalimage and patient anatomy using an image inversion prevention tag toensure the acquired medical image is not inverted with respect to thepatient anatomy.